1. Technical Field
This application relates to managing quality testing.
2. Description of Related Art
Computer systems may include different resources used by one or more host processors. Resources and host processors in a computer system may be interconnected by one or more communication connections. These resources may include, for example, data storage devices such as those included in the data storage systems manufactured by EMC Corporation. These data storage systems may be coupled to one or more servers or host processors and provide storage services to each host processor. Multiple data storage systems from one or more different vendors may be connected and may provide common data storage for one or more host processors in a computer system.
A host processor may perform a variety of data processing tasks and operations using the data storage system. For example, a host processor may perform basic system I/O operations in connection with data requests, such as data read and write operations.
A host system including a host processor may store and retrieve data using a storage device containing a plurality of host interface units, disk drives, and disk interface units. The host systems access the storage device through a plurality of channels provided therewith. Host systems provide data and access control information through the channels to the storage device and the storage device provides data to the host systems also through the channels. The host systems do not address the disk drives of the storage device directly, but rather, access what appears to the host systems as a plurality of logical disk units. The logical disk units may or may not correspond to the actual disk drives. Allowing multiple host systems to access the single storage device unit allows the host systems to share data in the device. In order to facilitate sharing of the data on the device, additional software on the data storage systems may also be used.
Tools are used for setting up tests for and measuring the performance of multi-device systems, such as a data storage system. As the size and complexity of computer systems increase, including both the number of host computers and the number and size of the disk drive elements, it becomes increasingly important to measure and understand the functions and parameters which affect or limit the performance of the system. The performance of the mass storage system can be typically measured in terms of input/output (I/O) response times, that is, the time it takes for a read or write command to be acted upon, as far as the host computer is concerned, by the disk drive controller system.
It is well known, in the field, to measure, usually using a single parameter, the instantaneous or average response time of the storage system. Typically, a host computer outputs one or more I/O requests to the disk drive controller, and then measures the time for a response to be received from the disk drive controller.
The performance of a large system, for example a large data storage system, is particularly difficult to measure. For example, more than one of the host computers, which connect to the disk drive controller(s), can operate at the same time, in a serial or in a parallel fashion. As a result, a plurality of disk drive elements, usually arranged in a disk drive array, operating in either an independent fashion, a RAID configuration, or a mirrored configuration, for example, can have a significant yet undetectable bandwidth or operational problem which cannot be addressed, or discovered, when commands are sent only from a single host computer.
When running the performance tests on a data storage system, one is usually interested in determining the maximum performance of components of the system, such as a host adaptor or SCSI adaptor (SA), a host adaptor port, a disk adaptor (or DA) etc., as well as the performance of the entire system.
A complex computing system such as a data storage system is typically the product of a large software development environment that relies on many developers, or software engineers, developing systems including many software elements. Each software element includes multiple objects, and is spread across many files. Typically, the files include source code, or human readable versions, of computer instructions, and are modifiable by the developers. As is known in the industry, the source code is compiled into object code and eventually into executable code, which is a set of instructions executable by a processor on the target machine for which the system was developed. The process of transforming the source code to the executable (running) system is known as a build, and involves a series of transformations from source code to executable code.
Software objects in conventional software systems include or inherit methods and attributes from other objects in the system. Inheritance is usually defined in terms of classes, which inherit other classes (or subclasses), resulting in a complex hierarchical or matrix pattern of inheritance, often called an inheritance lattice. Inheritance results in a dependency between the objects or classes, as changes to a particular class triggers a need to propagate the change to classes which inherit, or depend on, the changed class. Such dependencies are resolved by a build, which recompiles the affected entities to propagate the changes to dependent classes. In a large software system, the build can become substantial, and a modification to a particular class or object can have a rippling effect which traverses the inheritance lattice defined by the dependencies.
Computer programs often undergo several modification, development and improvement efforts. These development efforts may involve the modification of a program to improve its functionality and reduction or elimination of redundant variables and/or functions. Some of these development activities may involve the addition or removal of symbols and functions and the redefinition of factors. These development activities may or may not lead to functional changes in the compiled source code file and or header files of a program. Development activities that do not bring about any functional change in the compiled code may include removal of dead codes, restructuring header files, movement of functions from one file to another and creating self-compilable header files.
Customarily in the software development environment, multiple developers collaborate when writing source code. When collaborating, different developers independently work on separate modules or portions of the source code file. Source code management systems facilitate the collaborative authoring process by different developers. In most source code management systems, developers check-out a file or module, make changes, and check the file or module back into the source code management system. The newly checked-in file or module is assigned a version number, and that version is associated with an author. In addition to tracking the version number and author as part of the source code management system, the file or module may have additional metadata containing version and author information.
While working on a portion of source code, a developer may have questions regarding code written or changed by another developer in an earlier version of the code. It may be beneficial for developers to be able to identify the individual who introduced, modified, or deleted a specific portion of code in the current or a previous version of the source code. Most source code management systems provide some level of version control that indicates dates, times, and the author responsible for creating or modifying a version. Some source code management systems track changes by line, some track by module, and some source code management systems do not track changes at all.
The source code for software projects of all kinds (commercial, private, academic, governmental, open source, etc.) is routinely, universally, kept in specialized source code databases or repositories. Source code is represented as text files. Major software projects can have many thousands of such files, and scores of programmers making changes to various ones of these files. The primary purpose of source code repositories is to control and track changes to the source code and related (e.g., build, XML, configuration, documentation, test data, etc.) files. Various code control repositories are in widespread use (e.g. RCS (Revision Control System), SCCS (Source Code Control System), CVS (Concurrent Versions System), Subversion, CMVC (Configuration Management Version Control), Rational's ClearCase, Microsoft's Visual SourceSafe, etc.).
In general, as software applications have become increasingly complex, software development tools have been created to assist in the development of such applications. One such development tool, known as an integrated development environment (“IDE”), can assist the developer in understanding the structure, the dependencies, and the flow of complex applications. Examples of IDEs include Tornado® and SNiFF®, both available from Wind River Systems, Inc., as well as Visual C++®, available from Microsoft, C-Forge® available from CodeForge, VisualAge®, available from IBM, Jbuilder®, available from Borland, and SlickEdit®, available from MicroEdge.
In particular, software applications typically consist of many different files of source code that are then “built” together into a single application. Thus, development of software applications often involves developing these distinct source code files and revising those files over time. With complex software applications involving a high number of files, or in software development projects using multiple team members—or multiple teams—controlling access to those files and managing their development can be a very elaborate and complicated task.
Most IDEs in use today include some type of version control functionality coded into the IDE itself. Alternatively, some IDEs have been designed to function together with one specific version control tool. In those cases an adapter program provides an interface between the IDE and the separately coded version control tool. At least one IDE (SNiFF® version 3.0, available from Wind River) is compatible with several different version control tools. In SNiFF® version 3.0, the communication between the IDE and the version control tool is accomplished using various version control adapters, one for each version control tool supported by the IDE. Each version control adapter used with SNiFF® version 3.0 includes scripts that map the version control commands available in the SNiFF® menus with the corresponding version control commands of the version control tool to which it is adapted. The user dialogues, graphic user interface, and many of the version control and configuration management functions are performed by the IDE program itself. The version control adapters are relatively simple script files and are unable to make any but the simplest of decisions. Instead, the version control adapters act merely to dispatch commands and information between the IDE to the version control tool.
To date, at least several version control tools have been developed specifically to assist the software developer in controlling and managing revision information for each source code file. Version control is the process of managing and administering file revisions. CVS is an example of such version control tools. These version control tools help to manage multiple revisions of source code files by keeping a complete history of changes performed to the files. They allow the user to track changes made to the files and to return to a previous revision of a file. They may also keep track of who performed the revisions and when, and can “lock” a file to prevent two users from revising the file simultaneously. Typically, files are located in a repository that is directly accessed by the version control tool. As mentioned above, the file may be “checked out” of the repository by a developer, and changes may then be made to the file before it is checked back in. While the file is checked out in a locked state, other members of the development team may not make changes to it. When the file is checked back in it becomes accessible again to other members of the team, and the version control tool helps manage information about who checked out the document, when, and what changes were made. The tool may allow for the developer to save comments describing his changes and may assign a new name or version number to the file. Most of the version control tools offer these basic features in addition to other more advanced features.
Issue tracking systems and issue tracking databases are used to record and track both tasks that need to be completed for a project as well as open bugs and issues. Common examples of these systems include, but need not be limited to, Bugzilla and Team Track.
There are many well-known approaches to automated testing during the development of a software program or system. During the feature/test development phase, for example, a quality assurance (QA) engineer may devise a series of static test cases against which a system under development may be exercised. A set or collection of test cases is sometimes referred to a test suite.
In general, in order to help ensure the quality of software applications, quality assurance engineers use a variety of tools and procedures. For example, if an application has been modified, QA engineers test the application in order to ensure that additional bugs have not been introduced. Automated testing tools that perform regression testing are typically used to test the quality of a software application. WinRunner, by Mercury Interactive, is an example of an automated testing tool.
Regression testing is a quality control measure to ensure that newly modified code still complies with its specified requirements and that unmodified code has not been affected by any modifications. Regression testing is often used to the selectively retest a software application that has been modified to ensure that any bugs have been fixed. Furthermore, regression testing can be used to ensure that no other previously-working functions have failed as a result of the reparations and that newly added features have not created problems with previous versions of the software.
More particularly, regression testing is a way of testing software where the test verifies that the functionality of the software that worked previously continues to work. One form that regression testing takes is for the software to be executed and output collected from the software. The output is saved as the regression control. At a later point (often repeated at regular intervals), the software is executed again and further output collected. This second output is then compared to the regression control. If the comparison reveals differences between the second output and the regression control, a regression has been found. Subsequently, the tester has to determine whether that difference was due to an acceptable change in the software or whether an error was introduced.